Planning Referential Acts For Animated Presentation Agents

Computer-based presentation systems enable the realization of effective and dynamic presentation styles that incorporate multiple media. In particular, they allow for the emulation of conversational styles known from personal human-human communication. In this paper, we argue that life-like characters are an effective means of encoding references to world objects in a presentation. We present a two-phase approach which first generates high-level referential acts and then transforms them into fine-grained animation sequences. • effectively establish cross-references between presentation parts which are conveyed by different media possibly being displayed in different windows; • enable new forms of deixis by personalizing the system as a situated presenter. For illustration, let's have a look at two example presentations taken from the PPP system (Personalized Plan-Based Presenter, (RAM97)). In Fig. 1, a pointing gesture is combined with a graphical annotation technique using a kind of magnifying glass. 1 I n t r o d u c t i o n A number of researchers have developed algorithms in order to discriminate referents from alternatives via linguistic means (cf. (RD92)). When moving from language discourse to a multimedia discourse, referring expressions may be composed of several constituents in different media. Each constituent conveys some discriminating attributes which in sum allow for a proper identification of the referent. However, to ensure that a composed referring expression is intelligible, the system has to establish cohesive links between the single parts (cf. (AR94)). In this paper, we argue that life-like characters are particularly suitable for accomplishing referring tasks. For example, a life-like character can: • draw the viewer's attention to graphical object representations by pointing with body parts, and additional devices such as a pointing stick. • make use of facial displays and head movements as an additional means of disambiguating discourse references, Figure 1: Referring to Objects Using a Magnifying Glass The Persona provides an overview of interesting sites in the Saarland county by uttering their names and pointing to their location on a map. In addition, Persona annotates the map with a picture of each site before the user's eyes. The advantage of this method over static annotations is that the system can influence the temporal order in which the user processes an illustration. Furthermore, space problems are avoided since the illustration of the corresponding building disappears again after it has been 68 E. Andrd and T. Rist Figure 2: Establishing Cross-Media References described. The example also demonstrates how facial displays and head movements help to restrict the visual focus. By having the Persona look into the direction of the target object, the user's at tention is directed to the target object. Whereas in the last example, the pointing act of the Persona referred to a single graphical object, the scenario in Fig. 2 illustrates how cross-media links can be effectively built up between several illustrations. In this example, the Persona informs the user where the DFKI building is located. It utters: "DFKI is located in Saarbriicken" and uses two pointing sticks to refer to two graphical depictions of DFKI on maps with different granularity. As shown above, life-like characters facilitate the disambiguation of referring expressions. On the other hand, a number of additional dependencies have to be handled since a referring act involves not only the coordination of document parts in different media, but also the coordination of locomotion, gestures and facial displays. To accomplish these tasks, we have chosen a two-phase approach which involves the following steps: (1) the creation of a script that specifies the temporal behavior of the constituents of a referential act, such as speaking and pointing (2) the context-sensitive conversion of these constituents into animation sequences 2 Representation of t he M u l t i m e d i a Discourse A few researchers have already addressed referring acts executed by life-like characters in a virtual 3D environment (cf. (CPB+94; LVTC97)). In this case, the character may refer to virtual objects in the same way as a human will do in a real environment with direct access to the objects. A different situation occurs when a character interacts with objects via their presentations as in the example scenarios above. Here, we have to explicitly distinguish between domain objects and document objects. First, there may be more than one representative for one and the same world object in a presentation. For example, in Fig. 2, DFKI is represented by a schematic drawing and a colored polygon. Furthermore, it makes a difference whether a system refers to features of an object in the domain or in the presentation since these features may conflict with each other. To enable references to objects in a presentation, we have to explicitly represent how the system has encoded information. For instance, to generate a cross-media reference as in Fig. 2, the system has to know which images are encodings for DFKI. Inspired by (Mac86), we use a relation tuple of the form: (Encodes carrier info context-space) to specify the semantic relationship between a presentation means, and the information the means is to convey in a certain context space (cf. (AR94)). In our approach, the third argument refers to the context space to which the encoding relation corresponds to and not to a graphical language as in the original Mackinlay approach. This enables us to use one and the same presentation means differently in different context spaces. For example, the zoom inset in Fig. 1 is used as a graphical encoding of the DFKI building in the current context space, but may serve in another context as a representative building of a certain architectual style. In addition, we not only specify encoding relations between individual objects, but also specify encoding relations on a generic level (e.g., that the property of being a red polygon on a map encodes the property of being Planning Referential Acts for An ima ted Presentat ion Agents 69